docs: fix search ranking, add relevance smoke test and heading discoverability by kaviththiranga · Pull Request #847 · openchoreo/openchoreo.github.io

kaviththiranga · 2026-09-07T09:18:45Z

Problem

Searching the docs returned everything except the page you wanted. "Authorization" ranked the authorization overview at #57, and the top 20 results were just 5 pages — 4 of them AuthzRole CRD references.

Two causes in the Algolia index config:

searchableAttributes searched hierarchy.lvl0 (the sidebar category label) ahead of lvl1 (the page's H1). The API-reference category is labelled exactly Authorization, so its CRD pages beat the guide titled "Authorization in OpenChoreo".
attributeForDistinct was url, which includes the anchor — so one page contributed a record per heading and could fill every slot.

Fixed in the Algolia dashboard (not in this PR)

hierarchy.lvl0 moved after lvl6; attributeForDistinct → url_without_anchor with distinct: 2; both mirrored into the crawler's initialIndexSettings so a reindex can't revert them; /docs/category/* stubs no longer indexed.

Query	before	after	distinct pages in top 20
`authorization`	#57	#1	3 → 20
`rbac`	#21	#3	1 → 20
`observability`	#4	#5	4 → 20
`custom roles` / `cel conditions`	#1	#1	4 → 13 / 12 → 14

In this PR

scripts/search-smoke.mjs — replays the site's real search query and asserts the expected page ranks and that results span ≥8 distinct pages. No dependencies. Daily workflow + npm run test:search.
scripts/ALGOLIA.md, algolia-index-settings.json — the Algolia config only lived in a dashboard with no change history. Now reviewable and restorable.
Heading changes — search ranks headings far above body prose, so a term only in prose is nearly unfindable. This was reported for OpenTelemetry: the OTLP collector endpoints are documented, but no heading said "OpenTelemetry", "OTLP" or "collector". An audit of all 147 live docs files found 27 such terms; the top ones are fixed:

### Traces (OpenTelemetry) {#traces}
#### Instrumenting Applications with the OpenTelemetry Collector (OTLP) {#instrumenting-applications}
## Authenticated mirrors (imagePullSecrets) {#authenticated-mirrors}
## Configuring a ClusterSecretStore (Vault, AWS, GCP, Azure) {#configuring-a-clustersecretstore}
### GitOps installations (Flux, Argo CD) {#gitops-installations}

Anchors are pinned to their original IDs, so existing links keep working. Applied to both docs/ and versioned_docs/version-v1.2.x/ — the live /docs/* pages come from the versioned copy.

coderabbitai · 2026-09-07T09:18:54Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: bf3d89e1-f230-4b2e-a4a0-e2c7ef7479a8

📥 Commits

Reviewing files that changed from the base of the PR and between b9e97ac and 3cb3875.

⛔ Files ignored due to path filters (4)

versioned_docs/version-v1.2.x/platform-engineer-guide/air-gapped-installation.mdx is excluded by !versioned_docs/**
versioned_docs/version-v1.2.x/platform-engineer-guide/observability-alerting.mdx is excluded by !versioned_docs/**
versioned_docs/version-v1.2.x/platform-engineer-guide/secret-management.mdx is excluded by !versioned_docs/**
versioned_docs/version-v1.2.x/platform-engineer-guide/upgrades/v1.1-to-v1.2.mdx is excluded by !versioned_docs/**

📒 Files selected for processing (8)

.github/workflows/search-smoke.yml
docs/platform-engineer-guide/air-gapped-installation.mdx
docs/platform-engineer-guide/observability-alerting.mdx
docs/platform-engineer-guide/secret-management.mdx
docs/platform-engineer-guide/upgrades/v1.1-to-v1.2.mdx
package.json
scripts/ALGOLIA.md
scripts/search-smoke.mjs

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Summary

Summary by CodeRabbit

Documentation
- Added direct links to key sections covering authenticated mirrors, OpenTelemetry tracing, secret-management backends, and GitOps installations.
- Clarified the secret-management documentation by naming supported backends, including Vault, AWS, GCP, and Azure.
Search Reliability
- Added automated checks for common documentation searches to help ensure relevant results remain discoverable and correctly ranked.

Walkthrough

The PR adds a live Algolia search smoke test with scheduled and manual GitHub Actions execution. It documents the search configuration and adds explicit anchors to platform engineering documentation headings.

Changes

Search smoke testing

Layer / File(s)	Summary
Algolia search validation `scripts/search-smoke.mjs`	The new script queries the configured Algolia index with the site's filters. It checks expected result rank and page diversity, reports failures, and supports an index override.
Search smoke test wiring and configuration `.github/workflows/search-smoke.yml`, `package.json`, `scripts/ALGOLIA.md`	The `test:search` script and daily or manual workflow run the smoke test. The new documentation records the Algolia settings and validation rules.

Documentation anchors

Layer / File(s)	Summary
Platform guide deep links `docs/platform-engineer-guide/air-gapped-installation.mdx`, `docs/platform-engineer-guide/observability-alerting.mdx`, `docs/platform-engineer-guide/secret-management.mdx`, `docs/platform-engineer-guide/upgrades/v1.1-to-v1.2.mdx`	The documentation adds explicit anchors and expands selected headings for direct linking.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: 🟡 Moderate · up to 3cb38

This adds a daily search-validation workflow and documentation anchors. The workflow should explicitly restrict its token to read-only access before merge to avoid granting unnecessary repository-write capability.

Suggested reviewers: akila-i

Sequence Diagram(s)

sequenceDiagram
  participant GitHubActions
  participant SearchSmoke
  participant Algolia
  GitHubActions->>SearchSmoke: Run npm run test:search
  SearchSmoke->>Algolia: Submit golden query with facet filters
  Algolia-->>SearchSmoke: Return paginated hits
  SearchSmoke-->>GitHubActions: Report results and exit status

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 1 files. (7 skipped: 7…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the documentation search, relevance smoke test, and heading discoverability changes. It is concise and related to the pull request objectives.
Description check	✅ Passed	The description provides detailed problem context, explains the search-ranking changes, lists the in-scope files and heading updates, and documents the out-of-scope dashboard changes. It does not use …
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 1 files. (7 skipped: 7 unsupported.)

Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Signed-off-by: Kavith Lokuhewage <kaviththiranga@gmail.com>

kaviththiranga added 4 commits September 7, 2026 11:19

chore(search): add Algolia relevance smoke test and settings record

a0a9493

Signed-off-by: Kavith Lokuhewage <kaviththiranga@gmail.com>

test(search): allow per-case rank limits, hold observability to top 5

1fcf505

Signed-off-by: Kavith Lokuhewage <kaviththiranga@gmail.com>

docs(observability): name OpenTelemetry and OTLP in trace headings

dc73f85

Signed-off-by: Kavith Lokuhewage <kaviththiranga@gmail.com>

docs(search): name key technologies in headings, pin existing anchors

fe2ff6d

Signed-off-by: Kavith Lokuhewage <kaviththiranga@gmail.com>

kaviththiranga force-pushed the fix/search-ranking-smoke-test branch from 20abc85 to fe2ff6d Compare September 7, 2026 09:19

docs(search): drop unapplied settings snapshot, keep the rationale

3cb3875

Signed-off-by: Kavith Lokuhewage <kaviththiranga@gmail.com>

Navigation Menu

Uh oh!

docs: fix search ranking, add relevance smoke test and heading discoverability - #847

docs: fix search ranking, add relevance smoke test and heading discoverability#847
kaviththiranga wants to merge 5 commits into
openchoreo:mainfrom
kaviththiranga:fix/search-ranking-smoke-test

kaviththiranga commented Sep 7, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Sep 7, 2026 •

edited

Loading

Summary by CodeRabbit

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

kaviththiranga commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fixed in the Algolia dashboard (not in this PR)

In this PR

Uh oh!

coderabbitai Bot commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

kaviththiranga commented Sep 7, 2026 •

edited

Loading

coderabbitai Bot commented Sep 7, 2026 •

edited

Loading